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ABSTRACT: 

In order to realize asynchronous type system assuring the consistency of data with the function 
of disk subsystems without the need of introducing new software to a host unit and without the 
detenoration of the performance of a main center, in a remote copy system which copies the data 
to the disk subsystems of the remote center for duplicating the data in the disk subsystems of the 
main center, the disk subsystems give serial numbers and times to the data together with writing 
said data to the storage devices in the disk subsystem and transfer said data to the other disk 
subsystems, and the other disk subsystems arrange the two or more data in the sequence of the 
serial numbers, decide the oldest time among the latest time given to each of the disk 
subsystems communicating among the disk subsystems and the data given with the time not 
later than the decided oldest time are the objects of writing to each of the disk storage devices 
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DATA DUPLICATION SYSTEM 
AND METHOD BY MEMORY SUBSYSTEMS 

CLAIM(S) 

1) A data duplication method using a first memory subsystem group 
of multiple memory subsystems and a second memory subsystem group of 
multiple memory subsystem, in which to store the copies of data of the first 
memory subsystem group, comprising the following steps: 

by each memory subsystem in the first memory subsystem group, 
multiple data, each attached with a serial number and time, are input in each 
memory device of each memory subsystem and transferred to the memory 
subsystems in the second memory subsystem group via a transmission path; 

multiple data received by each memory subsystem in the second 
memory subsystem group are arranged in a sequence of serial numbers; 

through communication among the memory subsystems in the second 
memory subsystem group, the oldest time is selected out of the newest times 
provided by said memory subsystems; 
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the data attached with the time preceding the selected oldest time is 
input in each memory device of each memory subsystem. 

2) A data duplication method using memory subsystems, as cited in 
Claim 1 , wherein said transmission path for connecting said first memory 
subsystem group and said second memory subsystem group constitutes a 
storage area network (SAN). 

3) A data duplication method using memory subsystems, as cited in 
Claim 1 , wherein a time to be referred to by each memory subsystem in said 
first memory subsystem group is corrected by external time data. 

4) A data duplication method using memory subsystems, as cited in 
Claim 1 , wherein the memory subsystems in said second memory subsystem 
group are connected with a loop transmission path; each memory subsystem 
transfers to the adjacent memory subsystem the older time out of its own 
newest time and said received newest time, and the time that is transmitted 
and returned to its own memory subsystem is determined as said oldest time. 

5) A data duplication method using memory subsystems, as cited in 
Claim 1, wherein one of multiple memory subsystems in said second 
memory subsystem group is selected as a master memory subsystem; said 
newest time is notified to the master memory subsystem by each memory 
subsystem other than the master memory subsystem; the master subsystem 
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determines the oldest time out of its own newest time and the acquired 
newest time. 

6) A data duplication method using memory subsystems, as cited in 
Claim 1, wherein every data input in every memory subsystem in the first 
subsystem group is transferred, by every memory subsystem in said first 
memory subsystem in said first memory subsystem group, to one memory 
subsystem in said second memory subsystem group; the newest time is 

. selected out of the times provided by all the memory subsystems in said first 
memory subsystem group by one memory subsystem in the second memory 
subsystem group, and the oldest time selected out of the newest times is 
determined as a candidate for the oldest time. 

7) A data duplication method using memory subsystems, as cited in 
Claim 1, the memory device of said memory subsystem consists of multiple 
volumes, and when a volume in the first memory subsystem group to be 
copied from and a volume in the second memory subsystem to copy to 
constitute a volume pair, starting and stopping of the data transfer from the 
memory subsystems in the first memory subsystem group to the second 
memory subsystem group are controlled per a volume group consisting of 
multiple volume pairs. 



8) A data duplication system using memory subsystems, comprising a 
first memory subsystem group consisting of multiple memory subsystem 
and a second memory subsystem group consisting of multiple memory 
subsystems, wherein each memory subsystem in the first memory subsystem 
group has a means of inputting data in its memory device, a means of 
attaching a serial number and a time to the data, and a means of transferring 
the data attached with the serial number and time to the memory subsystem 
in the second memory subsystem via the transmission path; each memory 
subsystem in the second memory subsystem group has a means of arranging 
the received multiple data in a sequence of serial numbers, a means of 
determining the oldest time out of the newest times provided by the memory 
subsystems by communication among the memory subsystems in the second 
memory subsystem group, and a means of inputting the data attached with 
the time preceding the determined oldest time in each memory device of 
each memory subsystem. 

9) A data duplication system using memory subsystems, as cited in 
Claim 8, wherein said transmission path for connecting the memory 
subsystems in said first memory subsystem group and the memory 
subsystems in said second memory subsystem group constitutes a storage 
area network (SAN). 
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10) A data duplication system using memory subsystems, as cited in 
Claim 8, wherein each memory subsystem in said first memory subsystem 
group has a means of correcting said reference time for said times based on 
the externally supplided time data. 

1 1) A data duplication system using memory subsystems, as cited in 
Claim 8, wherein the memory subsystems in said second memory subsystem 
group are connected with a loop transmission path; each memory subsystem 
has a means of transferring to the adjacent memory subsystem the older time 
out of its own newest time and said received newest time and of determining 
the time transmitted and returned as the oldest time. 

12) A data duplication system using memory subsystems, as cited in 

Claim 8, wherein one of multiple memory subsystems in said second 
memory subsystem group is selected as a master memory subsystem; each 
memory subsystem other then said master memory subsystem has a means 
of notifying the master memory subsystem of said newest time; said master 
memory subsystem has a means of determining said oldest time out of its 
own newest time and acquired newest time. 

1 3) A data duplication system using memory subsystems, as cited in 
Claim 8, wherein each of multiple memory subsystems in said first memory 
subsystem group is structured to transfer each of the input data to one 
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memory subsystem in said second memory; said one memory subsystem in 
the second memory subsystem group has a means of selecting the newest 
time out of the times provided by said memory subsystems in the first 
memory subsystem group and a means of determining the oldest time out of 
the selected newest times as the oldest time candidate. 

14) A data duplication system by memory subsystems, as cited in 
Claim 8, wherein the memory device of said memory subsystem consists of 
multiple volumes; when a volume in the first memory subsystem group to be 
copied from and a volume in the second memory subsystem to copy to 
constitute a volume pair, the memory subsystems in the first memory 
subsystem group has a means of controlling the starting and stopping of data 
transfer to the second memory subsystem group per a volume group 
consisting of multiple volume pairs. 

1 5) A memory subsystem in a memory subsystem group consisting of 
multiple memory subsystems constituting a data duplication system, wherein 
said memory subsystem has a first means of inputting its externally received 
data in its memory device, a second means of attaching a serial number and 
time to said data and of transferring it to other memory subsystem, and a 
third means of arranging multiple data received from other memory 
subsystem in a sequence of serial numbers, and a fourth means of 
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determining the oldest time out of the newest times provided by other 
memory subsystems through communication among them; when said 
memory subsystem is in a local mode, it activates the first means and the 
second means, and in a remote mode, it activates the third means, the fourth 
means, and the first means; it inputs by the first means the data attached 
with the time preceding the determined oldest time. 

16) A memory subsystem constituting a data duplication system, as 
cited in Claim 15, wherein said memory subsystem operating in said local 
mode and said memory subsystem operating in said remote mode are 
connected with a storage area network (SAN). 

1 7) A memory subsystem constituting a data duplication system, as 
cited in Claim 15, wherein said memory subsystem has a means of 
correcting said reference time based on the externally supplied time data. 

18) A memory subsystem constituting a data duplication system, as 
cited in Claim 15, wherein said memory subsystems operating in said remote 
mode are connected with a loop transmission path, and each memory 
subsystem in said remote mode has a means of transferring the older time 
out of its own newest time and the received newest time to its adjacent 
memory subsystem and a means of determining the time transmitted and 
returned to its own memory subsystem as said oldest time. 
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19) A memory subsystem constituting a data duplication system, as 
cited in Claim 15, wherein one of said memory subsystems operating in said 
remote mode is selected as a master memory subsystem; each memory 
subsystem other than said master subsystem has a means of notifying said 
newest time to the master memory subsystem, and the master memory 
subsystem has a means of determining said oldest time out of its own newest 
time and the acquired newest time. 

20) A memory subsystem constituting a data duplication system, as 
cited in Claim 15, wherein multiple memory subsystems operating in said 
local mode are structured to transfer the data to one memory subsystem in 
the memory subsystem group operating in said remote mode; said one 
memory subsystem in the memory subsystem group operating in said remote 
mode has a means of selecting the newest time out of the times provided by 
the memory subsystems operating in said local mode and a means of 
determining the oldest time out of the selected newest times as said oldest 
time candidate. 

21) A memory subsystem constituting a data duplication system, as 
cited in Claim 15, wherein the memory device of said memory subsystem 
consists of multiple volumes; when a volume to be copied from, which is in 
the memory subsystem and operates in said local mode, and a volume to be 
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copied to, which is in the memory subsystem and operates in said remote 
mode, constitute a volume pair, the memory subsystem operating in said 
local mode has a means of starting and stopping the data transfer to the 
memory subsystem operating in said remote mode per every volume group 
consisting of at least one volume pair. 

DETAILED DESCRIPTION OF THE INVENTION 
(0001) . . 

(Field of Industrial Application) 

The present invention pertains to a memory subsystem for a computer 
to refer to and update data in, particularly to a data duplication system and 
its method for duplicating the data stored in the memory subsystem. 
(0002) 
(Prior Art) 

It is already known that there are companies and monetary institutions 
that have computer centers and back-up computer centers equipped with 
systems for backing-up data against earthquakes and disasters. In such a 
data back-up system, a primary computer center and a remote computer 
center are set up geographically apart by at least a distance of one hundreds 
kilometers - a few hundreds kilometers. Between disk subsystems of the 
primary center and of the remote center, data are stored and duplicated. In 
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one of its methods, data written in the disk subsystem of the primary center 
are transferred to the disk subsystem of the remote center in order to write 
the same data in the disk subsystem of the remote center. The systems 
having such a remote copying function have already been put to practical 
use. A remote copying function is primarily divided into two types: a 
synchronous type and an asynchronous type. 
(0003) 

A synchronous type refers to a process, wherein when a host computer 
(higher device) in the primary center commands its disk subsystem to update 
data (to input data) and when the object to which the command is given is a 
remote-copying function, the remote center, after having completed the 
updating and the remote copying of said data, which have been commanded 
to the disk subsystem of the remote center, reports the completion of the 
data-updating process to the higher device, of the primary center. In such a 
case, a physical distance between the primary center and the remote center 
has an impact on the performance of a data transmission path, generating a 
delay in transmission time. 
(0004) 

An asynchronous type refers to a process, wherein when the higher 
device in the primary center commands its disk subsystem to update data (to 



10 



input data), completion of the data updating is reported to the higher device 
as soon as the data has been updated even if the object to which the 
command is given is a remote-copying function, and the data updating 
(copying) in the disk subsystem at the remote center is executed 
asynchronously with the process at the primary center. By the asynchronous 
process, the data updating is completed within the processing time required 
in the primary center, so the time delay that is caused by storing the data in 
the remote center is not generated. Accordingly, when remote copying is 
performed between the subsystems at the remote sites, an asynchronous 
remote copying method will be more suitable than a synchronous method if 
high priority needs to be placed on avoiding an impact of the transmission 
time on the operation of the primary center. 
(0005) 

With the asynchronous type, the content of the subsystem in the 
remote center is not always consistent with that in the primary center. 
Therefore, if the primary center loses its function due to disasters, the data 
which are yet to be copied to the remote center will be lost. However, the 
access performance of the disk subsystem of the primary center can be 
improved to a level high enough not to use the remote-copying function. 
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(0006) 

The aforementioned prior art, however, comes with various problems 
since the remote-copying function is accomplished by an intermediate 
function of a higher device. 
(0007) 

"Preserving Data Consistency" 

When the remote copying is performed, the disk subsystem of the 
primary center and the disk subsystem of the remote center are connected by 
an independent communication link. More specifically, when remote 
copying is performed between multiple subsystems of the primary center 
and multiple subsystems of the remote center, multiple sets of subsystems, 
each set being connected with an independent communication link, are used. 
When the remote center backs up the primary center having multiple 
subsystems, there is a need of "preserving consistency of data," that is, a 
need of preserving a consistent sequence of data updating among multiple 
data subsystems. In asynchronous remote copying, it is unavoidable that 
updated data is copied to the remote center with a delay after the data 
updating has actually completed in the primary center. But, a sequence of 
updating must be consistent with that in the primary center. Even if the 
sequence of updating data is not always consistent, consistent data must be 
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at least recorded in the remote center when the user needs the data at the 

remote center. 

(0008) 

A data base generally consists of database body, various logging data 
indicating the history of data updating, and of control data, each being 
related to one another. The system is often designed so that the database 
body, various logging data, and control data are recorded in different 
subsystems, taking reliability into consideration. However, since the 
database body, various logging data, and control data are related to one 
another, the logging data and control data need to be added and updated 
when data is updated, while keeping their coordination in the system. The 
time interval between one updating and the subsequent updating in a 
sequence of data updating can be a few micron seconds if the updating is 
done in a short period of time in succession. If the sequence of updating is 
disrupted, the coordination of data to preserve the sequence of data updating 
is also disrupted, possibly destroying the entire database in the worst case. 
(0009) 

For example, it is possible that, even if the logging data is updated 
after the database body has been updated at the primary center, the updated 
logging data may arrive at the remote center before the logging data of the 
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database body does, depending upon the condition of said communication 
link constituting the remote copying system; then, the logging data will be 
added and updated before the database body is added and updated. If a 
disaster falls on the primary center under such a logically uncoordinated 
condition wherein only the logging data is added and updated but the 
database body is not added and updated, the data in the remote center will be 
useless. Therefore, data must be updated in the remote center exactly in the 
same sequence as the sequence in which data has been updated in the 
primary center. 
(0010) 

"Remote-Copying Function by Insertion of Higher Device" 

When asynchronous remote copying is performed in a general 
environment in which there are multiple disk subsystems in the primary 
center and in the remote center, there is a method wherein the data regarding 
the sequence of updating, such as a stamped time, is attached to data when 
the higher device of the primary center commands the disk subsystems to 
update data and the higher device of the remote center executes the data 
updating and copying based on said time data. For example, this technology 
is disclosed in Japanese Published Unexamined Patent Application 06- 
290125 (U.S. Patent 5446871). In Japanese Published Unexamined Patent 
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Application 06-290125, a remote copying operation is performed by 
inserting the higher device. More specifically, the cooperation between the 
operating system and subsystems of the primary center and the data mover 
software and subsystem of the higher device of the remote center 
implements the issuing and sending of the data regarding the sequence of 
updated data and the copying of the updated data by use of said data. 
(0011) 

(Problems of the Prior Art to Be Addressed) 

According to the prior art technology, asynchronous remote copying 
is executed while securing a sequence of updated data between the primary 
center and the remote center. In this method, however, the higher device and 
the disk subsystem both have to have a software application to execute the 
asynchronous remote copying function, and both sides have to cooperate in 
the operation. Then^ it becomes necessary for the user to reconsider the 
system design because of a need of introduction, installation, and inspection 
of the software application and of an increase in load on the CPU. 
Therefore, there is a problem that it takes long time and high cost for 
introducing the prior art technology. In addition, with the system wherein the 
higher device does not attach the data regarding the sequence of updating, 
such as a stamped time, or with the system not equipped with a mechanism 
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of matching the internal clocks in host computers, and particularly with an 
open system, there is no means of implementing said prior art technology. 
Therefore, in order to implement an asynchronous remote-copying function 
only by using the disk subsystem function, it is necessary to preserve, 
regardless of types of higher device and types of software application, the 
consistency in sequence of data updating only by using the disk subsystem 
function . Also, with the prior art technology, there was a problem that there 
was no means of preserving the consistency in sequence of updating among 
multiple disk subsystems when data requiring the consistency in sequence of 
updating are distributed to and stored in multiple subsystems. 
(0012) 

The objective of the present invention is to present an asynchronous 
remote-copying function that can secure consistency of data only by using 
the function of the subsystem without installing a new software application 
on the higher device and that can be introduced without reducing the 
performance of the primary center. 
(0013) 

(Mean to Solve the Problems) 

The present invention presents a data-duplicating method using a system 
comprising a first memory subsystem group consisting of multiple memory 
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subsystems and a second memory subsystem group consisting of multiple 
memory subsystems for storing the copy of data of the first memory 
subsystem group, said method being characterized by the following steps: 
data are input in the memory device of each memory subsystem of the first 
memory subsystem group while simultaneously transferring, via 
transmission path, said data to the memory subsystem of the second memory 
subsystem group by attaching a serial number and a time to said data; 
multiple data received by each memory subsystem of the second memory 
subsystem group are arranged in a sequence of serial numbers; every 
memory subsystem of the second memory subsystem group, by 
communicating with one another, determines the oldest time data out of the 
time data attached to the data and inputs the data attached with the time 
preceding said determined oldest time in the memory device of each 
memory subsystem. The present invention also presents a remote-copying 
system that functions according to the aforementioned method. 
(0014) . 

(Embodiment Example) 

The embodiment example of the system of the present invention is 
explained below with reference to the drawings. 



(0015) 

Fig. 1 shows the system structure wherein data is duplicated between 
two centers each having a computer. Multiple disk subsystems, 3-1,3-2, 
. . . , 3 - n, of primary center and the multiple disk subsystems, 7-1,7-2, 
. . . , 7 -n, of remote center 10 are connected without using the higher devices 
1 and 8, and both centers constitute the remote-copying system for 
duplicating data. The method of connecting the disk subsystems without 
using the higher devices 1 and 8 is not herein described in detail but, for 
example, a storage area network (SAN) can be used for the connection. 
(0016) 

In the primary center 9, the higher device 1 having a central processing 
unit (CPU) for data processing is connected to the disk subsystems, 3-1,3 
- 2, . . . , 3 - n, via the interface cable 2. 
(0017) 

The disk subsystem 3 - 1 is connected, via the interface cable 4 - 1 , to 
the disk subsystem 7 - 1 of the remote center. The disk subsystem 3 - 2 is 
connected, via the interface cable 4 - 2, to the disk subsystem 7-2 of the 
remote center. Likewise, the disk subsystem 3 - n is connected, via the 
interface cable 4 - n, to the disk subsystem 7 - n of the remote center. 
Hereinafter, one of the disk subsystems, 3 - 1, 3 - 2, . . . , 3 - n, or one of the 

18 



disk subsystems, 7 - 1 , 7 - 2, . . ., 7 - n, may be referred to as disk subsystem 
3 or disk subsystem 7. The same applies to other components as well. 
(0018) 

Also, the interface cables, 4 - 1 , 4 - 2, . . ., 4 - n, include an optic fiber 
link driven by the LED driving device and a fiber channel driven by an 
interface protocol by using an optic fiber cable. They may also include a 
telecommunication link represented by a T3 network or ATM network. 
Accordingly, a general fiber channel switch may be connected to some point 
along the interface cable between the disk subsystem 3 of the primary center 
9 and the disk subsystem 7of the remote center 10, or the interface cable 
may be connected to a T3 network, SONET network, and ATM network. In 
this example, the communication link is referred to as the interface cable 4 
provided that it can have any of these connection modes. 
(0019) 

In the remote center 10, one of the disk subsystems, 7-1,7 -2, 7 - n, to 
which the data is to be remote copied is, connected, via the interface cable 5, 
to one of the other subsystems in said center to which data is to be remote 
copied. In this example, the disk subsystem 7 - 1 is connected to disk 
subsystem 7-2, the disk subsystem 7 - 2 to disk subsystem 7-3, and disk 
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subsystem 7 - n to the disk subsystem 7 - 1 , forming a connected 

continuous loop. 

(0020) 

The higher device 8 is connected to the disk subsystems, 7-1,7-2,..., 
7 - n, via the interface cable 6 in the remote center, and it is a central 
processing unit that looks up and updates data in the disk subsystems, 7 - 1 , 
7 - 2, . . ., 7 - n. The higher device 8 can process data on behalf of the higher 
device 1 of the primary center 9 when the higher device 1 of the primary 
center 9 cannot perform its function due to failure or disaster. It also can 
executes a process different from that of the higher device 1 of the primary 
center 9 independently from the higher device 1. However, if the higher 
device 8 needs not process data in the disk subsystems, 7-1,7-2, 7- 
n, the higher device 8 is not necessary. 
(0021) 

Fig. 1 shows the system structure, wherein, in the case when the 
higher device 1 commands multiple disk subsystems, 3 - 1 , 3 - 2, . . . , 3 - n, 
to input data, the same data are also input in multiple disk subsystems, 7-1, 
7 - 2, 7 - n of the remote center 10, while preserving consistency in data 
updating with the updating in the primary center 9. The arrow in Fig. 1 



shows a flow of data, which the higher device has commanded to be input in 

the disk subsystems. 

(0022) 

Fig. 2 shows the internal structure of the disk subsystem 3 and of disk 
subsystem 7. The disk subsystem 3 comprises: interface controller 1 1 for 
receiving/sending a command and the data transferred from the higher 
device 1 and for connecting itself to other disk subsystem 7; cache memory 
12 for storing the data which is referred to or updated by the higher device 1 ; 
magnetic disk drive 13 as a recording medium for recording said data, 
control memory 14 for storing the control data for said data; the disk 
subsystem controller 17 for controlling each of these components. The disk 
subsystem controller 17 is operated by the program control that uses a 
microprocessor. 
(0023) 

The disk subsystem 7 has the internal structure similar to that of disk 
subsystem 3. Also the disk subsystem 3 and the disk subsystem 7 can be 
structured to have the same internal structure and the same function, and the 
disk subsystem functions as the disk subsystem 3 in local mode setting and 
functions as the disk subsystem 7 in remote setting. The functions in local 
mode and remote mode are explained below later. In addition, a memory 
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device other than the magnetic disk may take place of generai magnetic disk 
derive 13. For the disk subsystems 3 and 7, the ones generally called 
memory subsystems may be used. 
(0024) 

Each disk subsystem 3 is equipped with clock 15 and clock correction 
section 16. The clock correction section 16 frequently corrects the time of 
clock 15 based on the time data supplied by the time server located at a close 
but separate position from each subsystem to reduce the time difference 
between the time supplied from the time server and the time of clock 15 of 
each disk subsystem 3 to nearly a few micron seconds. In this context, the 
timer server has a function to receive radio wave containing the time data 
like a GPS or microwave clock does and a function to deliver the received 
time data to each disk subsystem 3. Or it can be the one wherein the time 
server has a function to generate and deliver the time to each disk subsystem 
without having the radio wave-reception function. If one of the disk 
subsystems 3 has the time server function, an independent time server is not 
needed. The time server 20 shown in Fig. 2 is located at a position close 
enough to the disk subsystem 3 and has GPS receiving section 21 for 
receiving data containing the time data from a satellite, a mechanism to seek 
the time from the received data, a mechanism to continue to generate the 
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time in the case when the time data cannot be received, and time delivery 

section 22 for delivering the time to disk subsystems 3. 

(0025) 

The frequency at which the higher device 1, which is generally referred 
to as a host computer, executes data updating in the disk subsystem 3 varies 
depending upon the system, but in the case when this updating is processed 
very frequently, the updating is performed successively at intervals of nearly 
micro seconds. The reason for accurately setting the clock 15 of every 
subsystem as accurately as a micro second range is to help the disk 
subsystem 3 accurately catch the time difference between the times attached 
to data even when the updating processes are executed successively at 
intervals of a few micro seconds and to catch the sequence of updating said 
data by using the times, when the data are updated. If the host computer 
commands the disk subsystem 3 to execute the updating at closer intervals 
than micro seconds, the clocks 15 of all the disk subsystems also must match 
the time among them by using a time unit smaller than said time interval. 
(0026) 

Fig. 3 shows the steps of process for duplicating data by the disk 
subsystem 3 and the disk subsystem 7. The initial condition prior to the 
following process presupposes that the previously added and updated data 



have already been copied between the disk subsystem 3 and disk subsystem 
7, therefore, the duplication of said data has already been completed. First, 
the higher device 1 commands the disk subsystem 3 to write data 
(hereinafter referred to as a write command) (Step 31). When the disk 
subsystem 3 receives this write command via the interface controller 1 1, the 
disk subsystem controller 17 of disk subsystem 3 begins to process said 
write command. The write command in this context includes the command 
to write data in cache memory 12 and the command to transfer the data to be 
written. The disk subsystem controller 17 of the disk subsystem 3 acquires 
the time when the write command is received from clock 15 (Step 32), stores 
the data in cache memory 12 (Step 33), and stores the control data of said 
data in control memory 14 (Step 34). This control data contains the address 
of the data to be written in the magnetic disk drive 13, the time at which the 
write command for said data is received, the serial number of the write 
command, and a pointer for the data in the cache memory 12. The received 
serial number of the write command is one number out of a series of 
numbers attached to the write commands received by said disk subsystem 3. 
Subsequently, the subsystem 3 reports the completion of the process of the 
write command to the higher device 1 (Step 35). The data stored in cache 
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memory 12 is later recorded on the magnetic disk drive 13, but this belongs 
to the prior art, therefore, is not explained in detail in the present invention. 
(0027) 

The disk subsystem controller 17, if it finds that the remote copying is 
not in a stop status (No response in Step 36) after referring to the control bit 
in the control memory 14. issues a write command to the disk subsystem 7 
connected to it, via the interface controller 1 1 and interface cable 4, 
asynchronously with processing of the write command from the higher 
device 1 (Step 37). In this context, this write command includes, like the 
write command received from the higher device 1, the command to write 
said data and the command to transfer said data, but this command further 
includes the time when said data is received and the serial number attached 
to the received data. If the. remote copying is in a stop status (Yes response 
in Step 36), the disk subsystem 3 stores the address data at which to write 
the data in the magnetic disk drive 13 in the control memory 14 of it own 
disk subsystem 3 (Step 38), and issues the write command to its partner 
subsystem 7 after the remote copying status returns to a normal status. 
(0028) 

The disk subsystem 7 receives the write command issued from the 
disk subsystem 3 via the interface controller 1 1 (Step 41). This write 
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command includes not only the data but the time when the disk subsystem of 
the primary center has received the data and the serial number with which 
the disk subsystem of the primary center has received said data. The disk 
subsystem 7 begins the process based on the received write command. The 
disk subsystem controller 17 of the disk subsystem 7 rearranges the received 
write commands according to a sequence of the serial numbers and checks 
out if there is a missing number, i.e., a missing write command in the 
sequence. Subsequently, the disk subsystem 7 stores the received data in 
cache memory 12 as "tentative data," stores the control data of said data in 
the control memory 14, and reports the completion of the data reception to 
its partner disk subsystem 3 (Step 42). 
(0029) 

Subsequently, the disk subsystem controller 17 of the disk subsystem 
7 selects a candidate of the correct time out of the time data contained in the 
received write commands and determines the correct time candidate in 
cooperation with the other disk subsystem 7 (Step 43). Determining the 
candidate for the correct time is to be explained later. When the correct time 
is thus determined, the data attached with the reception time preceding said 
time is stored in cache memory 12 as "correct data" (Step 44). The data 
stored in the cache memory 12 is recorded later on the magnetic disk. 
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(0030) 

Fig. 4 shows a flow of data in the process in which the data attached 
with the reception time and reception serial number are remote copied. Five 
sets of data to be written in the higher device 1, Dl, D2, D3, D4, and D5, are 
generated in this order in the higher device 1, and the higher device 1 
successively issues the write commands to the disk subsystem 3 - 1 and disk 
subsystem 3-2. The disk subsystem 3 - 1 receives data Dl, D3, and D5 
and attaches the reception serial number and the reception time to them as 
Sl/Tl, S2/T2, and S3/T5 in this order. On the other hand, the disk 
subsystem 3-2 receives data D2 and D4, and attaches the reception serial 
number and the reception time to them as S1/T2 and S2/T4 in this order. 
The disk subsystem 3-1, after storing the received data in the cache 
memory 12 in a sequence of the serial numbers, issues the write command of 
each data to the disk subsystem 7 - 1 . On the other hand, the disk subsystem 
3-2, after storing the received data in the cache memory 12 in a sequence 
of the serial numbers, issues the write command of each data to the disk 
subsystem 7-2. 
(0031) 

The disk subsystem 7- 1, upon receiving the data attached with the 
reception serial number and reception time, stores the data in cache memory 
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12 as a "tentative data." Subsequently, the data subsystem 7 - 1 and the data 
subsystem 7-2 compare the reception time of the newest data D5 out of the 
data received by disk subsystem 7-1 and that of the newest data D4 received 
disk subsystem 7-2. These data D5 and D4 are referred to as the correct 
time candidates. In this example, since the time T4 attached to the data D4 
is judged as being older than the time T5 attached to data D5, the time T4 
attached to data D4 is determined as the correct time, and the data D2 and 
data D4 attached with the time preceding this time are copied to cache 
memory 12 as the "correct data." 
(0032) 

As a result of above process, data D5 remains as the "tentative data," 
and data D5 is copied to cache memory 12 later than data D4 is. By 
preserving the sequential order of updating data D4 and data D5, these data 
can be recovered and their remote copying can be restarted when the remote 
copying is interrupted by a disaster. By contrast, if a disaster has occurred 
after data D5 is updated and copied prior to updating and copying of data 
D4, the data may be possibly lost in most cases, making it difficult to 
recover and copy the data. On the other hand, the sequence of updating the 
data Dl and D3 of disk subsystem 7 - 1 and data D2 and data D4 of disk 
subsystem 7-2, all of which are attached with the correct time, follows the 
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same sequence of updating data Dl and D3 of disk subsystem 3 - 1 and data 
D2 and D4 of disk subsystem 3-2; thereby preserving the sequence of 
updating. 
(0033) 

As explained above, the serial reception numbers indicate the 
updating sequence in one unit of the disk subsystems 3 and the updating 
sequence in one unit of the disk subsystems 7. This serial reception numbers 
are used to check out the missing data. On the other hand, the reception 
times indicate the time values in the data-updating sequence by multiple 
units in disk subsystem 3 and are used for the multiple units of the disk 
subsystem to determine the correct time. 
(0034) 

Fig. 5. shows steps of the process of disk subsystem 7 in determining 
the correct time. The disk subsystem controller 17 of disk subsystem 7 that 
issues a request for determining the correct time out of the correct time 
candidates selects the candidate in its own device (Step 51); more 
specifically, it selects the write command attached with the newest reception 
time out of the write commands arranged in a sequence of the serial 
reception numbers of data; subsequently, it commands the adjacent disk 
subsystem 7 connected with interface cable 5 to determine the correct time 
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candidate (Step 52). This command includes the request indicating the 
request for determining said candidate, the time attached to the correct time 
candidate, and the product number of said disk subsystem 7 (product No.). 
At this time, the time when the request is made is stored in its own control 
memory 14. 
(0035) 

The adjacent disk subsystem 7, upon receiving this command, 
determines whether the product number in the command agrees with the 
product number of it own device (Step 53); if the product number does not 
agree with that in its own device (No response in Step 53), it selects the 
correct time candidate by the above process in its own device (Step 54); 
subsequently, it compares the time of the received correct time candidate 
with the time of the correct time candidate in its own device (Step 55). If the 
time in its own device is older (Yes response in Step 55), the adjacent disk 
subsystem 7 replaces the determined time of the correct time candidate and 
the product No. with the time of the correct time candidate and the product 
No. in its own device (Step 56). If the time in its own device is newer (No 
response in Step 55), it moves to the step 57; subsequently, it commands the 
adjacent disk subsystem 7 connected by interface cable 5 to determine the 
correct time candidate between the replacement time in its own device and 
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the received correct time candidate (Step 57). It also stores the time of this 
request in its own control memory 14. After.the process in Step 57, it returns 
to the Step 53. 
(0036) 

The adjacent disk subsystem 7, upon receiving said command 
requesting to determine the correct time, determines the correct time 
candidate, for which the request was made, as the correct time (Step 58) 
when its product No, in the command agrees with the product No. of its own 
device (Yes response in Step 58). More specifically, the correct time 
candidate is selected by every device of the disk subsystem 7, and the oldest 
time among their candidates is determined as the correct time. 
(0037) 

Fig. 6 shows a flow of the process by the disk subsystem in notifying 
the correct time. The disk subsystem 7 that has determined the correct time 
stores the data attached with the time preceding the correct time as the 
"correct data" in cache memory 12 (Step 61); subsequently, it issues a 
command notifying the adjacent disk subsystem 7 of the determined correct 
time (Step 62). This command contains the request indicating the notice of 
the correct time, the determined correct time, and the product No. of said 
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disk subsystem 7. At this time, it also stores the time when the command 

was issued in the control memory 14. 

(0038) 

The adjacent disk subsystem 7, receiving this command, determines 
whether the product No. contained in the command agrees with that of its 
own device (Step 63); if the product No. does not agree with that of its own 
device (No response in Step 63), the adjacent disk subsystem 7 stores the 
data attached with the time preceding to the correct time that was received 
from disk subsystem 3 in cache memory 12 as the "correct data" (Step 64); 
subsequently, it sends the received command to the adjacent disk subsystem 
7 to notify the correct time (Step 65), and returns to the Step 63. 
(0039) 

The adjacent disk subsystem 7, upon receiving the command notifying 
the correct time, deletes (Step 66) from the control memory 14 the 
unnecessary data, such as the product No. out of the data attached with the 
time preceding the correct time when the product No. in the command 
agrees with the product No. of its own device. As a result of the above 
process, all the write commands to which all the subsystems 7 have attached 
the time preceding the correct time are provided to the data to be updated in 
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the disk subsystem 7, and a new command after the correct time remains as 

the subsequent correct time candidate to be selected. 

(0040) 

In one method of activating the request for determining the correct 
time, the disk subsystem to be activated first is preselected, and hereinafter, 
the disk subsystem 7 that has determined the correct time activates the 
request for determining the next correct time. Or one unit of the disk 
subsystems 7 may activate said request periodically. Either method can be 
used as long as the step of determining the correct time and the determined 
correct time are notified periodically in the system. 
(0041) 

In the above example, the manufactured product No. is used as an 
identifier of each disk subsystem 7, but any identifier can be used as long as 
it can identify the every disk subsystem 7 of the remote center. 
(0042) 

The disk subsystem 7, if the correct time determining and requesting 
command which it has issued did not return or the correct time notice was 
not received within the prescribed time period, determines that the correct 
time was not determined or the correct time notice was not completed for 
some failure, sets the control bit in the control memory 14 to the remote 
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copy stopping status, and stops the remote copying. When the remote 
copying cannot be continued because of a failure on the side of disk 
subsystem 7, it stops the remote copying, and notifies the partner subsystem 
3 of said stop of remote copying. The disk subsystem 3, upon receiving this 
notice, sets the control bit in the control memory 14 to the remote copy 
stopping status, stops the remote copying, keeps the write command 
received from the higher device from being transferred to the disk subsystem 
7. Also, when the remote copying cannot be continued because of a failure 
on the side of disk subsystem 3, it likewise stops the remote copying and 
notifies its partner disk subsystem 7 of said stop of remote copying. Or in 
the case when the disk subsystem 3 and the disk subsystem 7 have detected 
the communication unable due to failure of interface cable 4, they likewise 
stop the remote copying. In addition, at a proper point in time, they can 
restart the remote copying by changing the control bit in the control memory 
14 by the command from the service processor panel 1 8 of disk subsystem 3 
or disk subsystem 7. 
(0043) 

In addition, a temporary stop and restart of remote copying can be 
designated by a volume pair unit, which is a minimal control unit of the 
recording region in the disk subsystem when the remote copying is preset. 



The volume pair refers to a pair of volumes formed by a volume in the disk 
subsystem 3 to be copied from and by a volume in the disk subsystem 7 to 
be copied to. At least one volume pair is defined as one volume group, and 
the status can be changed by a volume group unit. The data updating inside 
the volume group does not affect other volume groups. In this case, the 
control bit in the control memory 14 is installed per a volume pair or volume 
group. The disk subsystem 3 refers to the control bit in the control memory 
14 that corresponds to the volume or volume group in Step 36 to determine 
whether the remote copying of the data is stopped or not. 
(0044) 

Accordingly, in the system wherein all the business data stored in the 
multiple disk subsystems are constantly duplicated by remote copying, when 
some data at the operation completion point needs to be used, said data at the 
operation completion point will be put on hold in the disk subsystem if the 
remote copying by the volume pair or volume group, in which said operation 
data has been stored at the operation completion point, is temporarily 
stopped. Therefore, the operation can be continued at the remote center. 
(0045) 

Once the temporary stop is released, the disk subsystems 3 and 7 
restart the remote copying process and transfer to the disk subsystem 7 the 
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write command of the data written only in the disk subsystem 3 of the 
primary center during the temporary stop as well as the data which is not yet 
defined as the "correct data" in disk subsystem 7. Thus the data is defined 
as the "correct data" defining process can be restarted. By this, a normal 
remote copying status is resumed. 
(0046) 

In the system structure shown in the above embodiment example, the 
disk subsystem 3 of the primary center 9 and the disk subsystem 7 of the 
remote center 10 are connected in one-on-one relationship. As the other 
embodiment example, it is also possible to construct the system wherein 
multiple disk subsystems 3 are connected to one unit out of disk subsystems 
7, and remote copying is performed between this disk subsystem 7 and said 
multiple disk subsystems 3. In this structure, the disk subsystem 7, upon 
receiving data in Step 41, rearranges the received write commands in a 
sequence of the serial numbers of the received data by using the product 
numbers of the disk subsystems, selects the newest time out of the times 
attached to the write commands of the disk subsystems 3, compares the 
times selected by the disk subsystems 3, selects the oldest time out of them 
as the correct time candidate, and determines the correct time. 
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(0047) 

Moreover, as another embodiment example, it is possible to construct 
the system structure, wherein one unit out of the disk subsystems 7 of the 
remote center is used as a master disk subsystem (generally referred to as a 
master memory subsystem), and all the disk subsystems 7 other than the 
master disk subsystem in the remote center 10 are connected to the master 
subsystem. In this structure, the master disk subsystem acquires by inquiry 
the correct time candidate retained in the other disk subsystems. 
Subsequently, the master disk subsystem compares the acquired correct time 
candidate with the its own correct time candidate, determines the oldest time 
as the correct time, and notifies the other disk subsystems of the remote 
center of the determined correct time; thus, each disk subsystem 7 
determines the "correct data." 
(0048) 

In this embodiment example, since the disk subsystem 7 of the remote 
center 10 determines the correct data based on the correct time candidate and 
the correct time, asynchronous remote copying can be embodied by the disk 
subsystems while preserving the sequence of data updating without inserting 
the higher device, without periodically interrupting the remote copying, 
without installing a gateway subsystem for preserving the data updating 
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sequence on the primary center 9 and remote center 10, and without 
reducing the process performance of disk subsystem 3 of the primary center 
9. 

(0049) 

According to the embodiment examples of the present invention, the 
data updating sequence is ensured by using the time acquired from the clock 
contained in the disk subsystem, so asynchronous remote copying can be 
embodied while preserving the sequence of data updating without inserting 
the higher device 1, even in the system wherein the time data is not attached 
to the data transferred from the higher device 1 to the disk subsystem 3. 
(0050) 

In addition, even if the function of the primary center is broken down 
because of a disaster or failure of the primary center, data for which the data 
updating sequence is preserved is stored in the disk subsystem 7 of the 
remote center 10. These processes are carried out only by the functions of 
the disk subsystems 3 and 7 without imposing a burden on the processing 
capability of the higher device 1 . In the case when a disaster falls on the 
primary center, a recovery operation such as restarting of a job, can be 
carried out by using the data in the disk subsystem 7, so the operation can be 
restarted. 
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(0051) 

(Advantage of the Invention) 

As explained above, an asynchronous remote copying system, which 
can be easily introduced and does not reduce the processing performance of 
a host computer of a primary center, can be implemented only by the 
functions of memory subsystems, while ensuring the data updating sequence 
to the extent that the users expect, without introducing a new software 
application into the higher device. 

BRIEF DESCRIPTION OF THE DRAWINGS 

Fig. 1 shows the entire structure of the remote copying system as one 
embodiment example of the present invention. 

Fig. 2 shows the internal structure of the disk subsystem in the 
embodiment example. 

Fig. 3 shows a flow of the steps of data duplication between the disk 
subsystem 3 and the disk subsystem 7 in the embodiment example. 

Fig. 4 illustrates one example of a data flow in which the data, each 
being attached with a reception time and a serial reception number, are 
remote copied. 

Fig. 5 shows the steps of a process in which the disk subsystem 7 
determines the correct time in the embodiment example. 



Fig. 6 shows the steps of a process in which the disk subsystem 7 
notifies the correct time in the embodiment example. 
1. higher device 

3. disk subsystem 

4. interface cable 

5. interface cable 
7. disk subsystem 

9. primary center 

10. remote center 

15. clock 

16. time correction section 



